The document itself is good.

At the same time, what do you think about rigorously dropping everything what's flaky? Flaky tests are a nuisance to everyone who invests good work to get changes through and have the side-effect of wearing down motivation to painstaking.

As long as we don't mistake flaky Hydra for flaky tests, this would absolutely be a win.

We could perhaps move them to a flaky attrset, so that they can be fixed and moved back when they have a record of being reliable again.

Reasons to prefer dropping over moving:

• less is more
• satisfying

Reasons to prefer moving:

• the state of the test remains visible, so that someone can pick it up and debug it
• some tests need to evolve to a non-flaky state, as race conditions are discovered
• no loss of coverage
• Hermit

Hermit emulates the scheduler deterministically. This should let us find debuggable counterexamples, I think. Might be too new/experimental, and might not support KVM. We'll want to run it only on the flaky tests, because it is slower by an order of magnitude, but NixOS isn't poor so that's not a reason to avoid it.

So yeah, I think moving those tests would be great!

Having different sets of which only one is really mandatory for contributors to satisfy sounds like a great improvement, yes.

Opened #216828 for this.

FWIW, @JulienMalka wanted to explore Hermit usage in NixOS tests, I don't know if he has something to share, but just mentioning him as it can be interesting.

Indeed I think it'd be very interesting to deep a little further into Hermit, see what improvements in test reproducibility we can get and at what cost. I'll certainly work on that subject soon, available to chat more about it.

What parts of the nixos test driver stack would be touched by using hermit? My gut feel says that this would be "just" configuring the guest vm's nixos config with kernel patch + config properly, is that about right?

Ideally Hermit supports KVM, so that we only have to wrap the top level invocation in nixos/lib/testing/run.nix. Otherwise, we have a qemu invocation in nixos/modules/virtualisation/qemu-vm.nix, which is included (a bit indirectly) from nixos/lib/testing/nodes.nix. Configuring the kernel works the way it normally does in NixOS, I think.